Latent Class Mixture Models of Treatment Effect Heterogeneity for Post Hoc Subgroup Analysis

In randomized experiments, it is often of interest to characterize treatment effect heterogeneity in terms of baseline covariates. Usually, the aim is to identify subpopulations likely to have particularly positive or negative (or neutral) responses to treatment. The process of searching for such subpopulations after the completion of an experiment (without pre-specifying which subpopulations will be considered as candidates) is called ’post hoc subgroup analysis’. It is a controversial practice. Concerns about data dredging and multiple comparisons (Rothwell, 2005) have led many authors to advise against reporting results from post hoc subgroup analyses at all. However, it is our view that post hoc analyses can produce informative insights that would be unlikely to arise from limited pre-registered comparisons. Here we illustrate an approach in which identification of special subgroups is one byproduct of fully modeling treatment effect heterogeneity more generally. By placing shrinkage priors on relevant parameters and applying cross validation based tools for model evaluation and comparison, we minimize data dredging concerns and provide a mechanism for gauging confidence in the substantive implications of model results.